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This paper studies the concept of color semantics by modeling a dataset of magazine cover designs, evaluating the model via 
crowdsourcing, and demonstrating several prototypes that facilitate color-related design tasks. We investigate a probabilistic 
generative modeling framework that expresses semantic concepts as a combination of color and word distributions —color- 
word topics. We adopt an extension to Latent Dirichlet Allocation (LDA) topic modeling called LDA-dual to infer a set of 
color-word topics over a corpus of 2,654 magazine covers spanning 71 distinct titles and 12 genres. While LDA models 
text documents as distributions over word topics, we model magazine covers as distributions over color-word topics. The 
results of our crowdsourced experiments confirm that the model is able to successfully discover the associations between 
colors and linguistic concepts. Finally, we demonstrate several simple prototypes that apply the learned model to color palette 
recommendation, design example retrieval, image retrieval, image color selection, and image recoloring. 
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1. INTRODUCTION 

Although researchers have endeavored to understand the role of color perception in design of human 
computer interactions, in user engagement and first impression [Lindgaard et al. 2006; Reinecke 
et al. 2013], trust and credibility [Lindgaard et al. 2011; Pengnate and Antonenko 2013], perfor¬ 
mance [Moshagen et al. 2009; Sonderegger and Sauer 2010], aesthetics and usability [Lavie and 
Tractinsky 2004; Hassenzahl 2004], and culture [Cyr et al. 2010; Reinecke and Bernstein 2011; 
Reinecke and Gajos 2014], understanding the cognitive aspect of color merits further investigation. 
It is more evident in the recent studies [Derefeldt et al. 2004; Skorupski and Chittka 2011] that col¬ 
ors are cognitive: Perceived by our visual system, colors are classified at higher levels of abstraction 
into verbal and semantic categories [Humphreys and Bruce 1989; Barsalou 1999] by a visual task 
such as color categorization [Derefeldt et al. 2004]. Nevertheless, in practice, for instance in visual 
design, color combinations are designed not only to be appealing, but also to be silent salesper¬ 
sons that communicate with the audience [Eisemann 2000; Frascara 2004; Newark 2007; Samara 
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Fig. 1: Application of color semantics in color palette selection, and design example retrieval. See 
Sec. 9. 


2007]. In schools of design, students learn creation and usage of color palettes to convey emotions 
and ideas [Whitfield and Wiltshire 1982; Ou et al. 2012]. Devising generative models which can 
associate color combinations with linguistic concepts, based on the work of designers, might en¬ 
able more meaningful user interactions for communicating with colors through color design and 
cognition. 

Color semantics in computer science and engineering has been observed as a gap [Smeulders et al. 
2000; Sethi et al. 2001; Mojsilovic and Rogowitz 2001; Liu et al. 2007]. Nevertheless, in terms of 
associating emotions and moods with colors, there is a recent body of work [Csurka et al. 2010; Solli 
and Lenz 2010; Murray et al. 2012]. In current frameworks, the semantics are labels provided by 
human inputs, leading to a manual collection and a limited number of semantics. However, the task 
of automatically inferring the linguistic concepts from the data is still unsolved. Another limitation 
is that these semantics are defined in bipolar scales, such as happy-sad. This limits the notion of 
color semantics to the opposites. In other words, more abstract concepts, such as management will 
not be captured. 

In this paper, we investigate a probabilistic generative modeling framework to associate color 
combinations with linguistic concepts from professional design examples. The model is an adapta¬ 
tion of LDA-dual, an extension of the popular Latent Dirichlet Allocation (LDA) topic model [Blei 
et al. 2003] to jointly link colors and words, and infer color-word topics. Originally, the LDA-dual 
model was proposed by Shu et al. [Shu et al. 2009] for complete entity resolution of names in the 
bibliography of scientific publications. Because LDA is both an inference mechanism and a gen¬ 
erative model, our framework utilizes both these features to automatically infer semantics, and to 
enables more meaningful user interactions for color selection and design. As one application. Fig. 1 
shows a system for finding design examples based on color semantics. 

In visual design, designers use both colors and words as design elements to carry their messages in 
a unified form of design. This observation suggests that each design’s theme might be a combination 
of words and color distributions; and each design may include a proportion of various themes. A 
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similar intuition is argued in LDA [Blei et al. 2003], for textual documents. That is, a document can 
be viewed as a proportion of different topics, where each topic is a histogram of words. Although 
one may adopt LDA to capture the underlying word topics or color topics in design examples, our 
aim is to simultaneously link both of these topics via our framework. The model produces combined 
color-word topics. The word topics are multinomial distributions on words; and the color topics are 
histograms of colors. Later, to facilitate the user interactions, we visualize the word topics as word 
clouds, and discretize the resulting color histograms to yield a mapping to 5-color palettes. 

Seeking good examples of visual design, we collected a relatively large corpus of magazine cov¬ 
ers. We chose magazine covers for two reasons: First, by using a variety of data sources including 
the Internet, archives of our own university library, and widely accessible newsstands, we were able 
to create a dataset of 2,654 covers from 71 titles and 12 genres. Second, designers spend many 
days conceptualizing and creating a magazine cover that attracts the audience at first glance, while 
competing with other magazines on a newsstand [Foges 1999]. In this process, designers carefully 
choose a color palette for the cover based on the topic and cover lines of the issue in order to convey 
a very specific concept. 

To verify whether or not users agree with the associations between color combinations and lin¬ 
guistic concepts produced by the model, we conduct a crowdsourced experiment. In particular, users 
are shown discretized color palettes and a choice of word clouds and are asked to select the most ap¬ 
propriate word cloud for the given color palette. Based on the user feedback, we infer the relevance 
of the color palette to the word cloud in the experiment. 

The flow of this paper is as follows. In Sec. 2, we present a study on color semantics in both 
theoretical and practical aspects. This section is then followed by Sec. 3 where we review related 
work in computer science and engineering. In Sec. 4, we discuss how we preprocess the data, and 
define vocabularies for words and the basis that we use to represent colors in our system. We then 
discuss the inference and generative mechanisms in the modeling framework in Sec. 5. For ease 
of visualization, in Sec. 6 we represent the word topics as word clouds, and discretize the result¬ 
ing color histograms to yield a mapping to a pool of 5-color palettes. We then discuss the design 
of the crowdsourcing experiments in Sec. 7, and analyze the crowd responses in Sec. 8. In Sec. 9, 
we suggest a number of applications for color semantics to demonstrate how semantics can enable 
more meaningful user interactions. We specifically discuss color palette selection and design exam¬ 
ple recommendation, image retrieval, color region selection in images, and pattern colonization in 
image recoloring. We conclude this paper and suggest a number of interesting avenues for future 
work in Sec. 10. 

2. THEORY OF COLOR SEMANTICS 

Colors are colorful: more than physical attributes of objects, colors are cognitive and have names, 
meanings, and values. In the following subsections, these aspects of colors are briefly reviewed, and 
the need to understand the role of color semantics in analysis and synthesis of visual designs in HCI 
is elucidated. 

2.1. Color Cognition 

While color perception is a rich area of research, color cognition is relatively new [Pylyshyn et al. 
1999]. That is, although color is perceived as the result of wavelength discrimination, it is also 
cognitive [Skorupski and Chittka 2011]. Derefeldt et al. [Derefeldt et al. 2004] elucidate that “cog¬ 
nitive” means that after visual perception, color is classified to a higher level of abstraction into 
verbal and semantic categories [Humphreys and Bruce 1989; Barsalou 1999] by a visual task such 
as color categorization. 

Heer and Stone [Heer and Stone 2012] suggest that when this categorization capability (in the 
concept of color naming) is deployed in user interfaces that model human category judgements, 
it might demonstrate more meaningful and novel user interactions. What the boundaries are for 
color naming, and how this concept can be compared with color meanings has not yet received the 
attention of scientists. 
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2.2. Color Naming 

The earliest effort to understand linguistic categorization of colors is perhaps the work of Berlin 
and Kay for elucidating that color naming is universal and related to evolution [Berlin 1991]. Be¬ 
fore their work, most linguists believed that the concept of color naming —associating names with 
colors— is not universal. As Kay and McDaniel [Kay and McDaniel 1978] summarize, the two 
main misconceptions about color naming among linguists were that color naming is a matter of 
cultural relativism and that semantic primes in languages are discrete entities. The latter doctrine 
was not able to address compound terms, e.g. green-blue. This inadequacy motivated Kay and Mc¬ 
Daniel to describe color categories as continuous functions using fuzzy set theory. The universality 
claim of color naming [Berlin 1991] does pertain under some conditions, as explained by Palmer 
in his book Vision Science: Photons to Phenomenology [Palmer 1999]. Although color naming as¬ 
sociates words with colors, there are other trends of research that investigate meanings and higher 
level abstractions/semantics of colors. 


2.3. Color Meanings and Semantics 

The first systematic approach to quantifying “meanings” of linguistic concepts comes from mea¬ 
surement of meaning by Osgood [Osgood 1952; 1957]. Osgood proposes a semantic space based 
on pairs of polar terms, such as happy-sad or kind-cruel, and terms it a “semantic differential” as 
an “objective index of meaning” [Osgood 1952]. Osgood then studies cross-cultural generality of 
his semantic space and finds significant similarities between different primitive cultures [Osgood 
I960]. Later, Osgood compiled a 620-concept Arte of Affective Meanings, explained in [Osgood 
1971] with a report of cross-cultural studies in 23 cultures. Adams and Osgood [Adams and Osgood 
1973] investigate the 8 color concepts of Osgood’s Atlas —color (vs monochrome), white, grey, 
black, red, yellow, green, and blue — among 20 countries and report that for instance, the relative 
affective meaning of red is strong and active. Another similar attempt is due to Wright and Rain¬ 
water [Wright and Rainwater 1962], yet with a more visual communication language perspective in 
color meanings and connotations. This is where a set of more design-oriented color semantics such 
as “elegant” or “showiness” emerges. 

A breakthrough in design-based color semantics is the study by Kobayashi [Kobayashi 1981]. 
Kobayashi defines a meaning scale to relate “worlds of people and objects with worlds of colors” and 
terms it Color Image Scale [Kobayashi 1991]. This scale is comprised of two bipolar dimensions: 
warm-cool and soft-hard. Kobayashi’s scale has significant contributions to visual design: It is based 
on 3-color combinations (3-color palettes) rather than single color patches (swatches). It also relates 
these 3-color combinations with two levels of abstractions, one with 180 semantics, and the other 
with 15 higher level semantics. Because of these characteristics, Kobayashi’s scale is taught in many 
color courses in schools of design (e.g. see [Green-Armytage 2006]). Later cross-cultural studies 
acknowledge the universality of Kobayashi’s bipolar scales [Ou et al. 2004a; 2004b; Ou et al. 2012]. 
A similar effort in association of colors and words was undertaken by Sivik as mentioned in [Hardin 
andMaffi 1997]. 


2.4. Color Semantics, Emotions, and Preferences 

While investigating meanings of colors, some researchers have attempted to understand the emo¬ 
tions and moods evoked by colors and how feelings about colors can infiuence our performance 
(e.g. see [Thuring and Mahlke 2007] and [Lin et al. 2013a]). From a psychological point of view 
for instance, Crozier [Crozier 1996] argues that theories of preference based on innate and learned 
reactions should be considered while studying color meanings. Crozier summarizes that red, for 
instance, has the innate mood of the alert signal and also is involved in sexual behaviour in many 
species; or white is learned to be associated with purity in some cultures. Although Crozier ac¬ 
knowledges that the like-dislike bipolar scale in prior studies is a useful measure to investigate color 
moods, he argues that meanings of individual colors should be considered within the context (syn- 
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tax, semantics, and culture) in which they are examined, also adding other factors such as age and 
gender. 

Later, Ou et al. [Ou et al. 2004a; 2004b; 2004c] studied color emotions and color preferences to 
clarify the relations between them. Although their studies agreed with prior work in defined scales 
of warm/cool, heavy/light, and active/passive, they found notable differences in the like-dislike scale 
between their two groups of participants, who were Chinese and British. They also found a tendency 
for their participants to prefer color combinations that hold opposite emotions. 

2.5. Color Semantics and Cross-cultural Considerations 

Hutchings summarizes that “The Principle of Adaptation of Ideas accounts for regional variations 
in colour folklore. This embodies a Darwinian-type principle of behavior, that is, ‘to survive within 
a community, a belief must have relevance to that community’” [Hutchings 2004]. Nevertheless, it 
is observed that in the information era, common sense about colors is increasing [Carroll 2007]. 
Such a cultural problem in HCI can be thought of as “minor science” as [Ghassan and Blythe 2013] 
suggest. 

In a recent cross-cultural study of color emotions of 190 two-color combinations from inhabitants 
of eight countries: Argentina, France, Germany, Iran, Spain, Sweden, Taiwan, and UK, Ou et al. [Ou 
et al. 2012] report consistency for warm/cool, heavy/light, and active/passive. This study also points 
out some inconsistency for like/dislike. For instance, Argentinian participants preferred more gray¬ 
ish colors in contrast to other participants. However, additional cross-cultural studies could shed 
light on our knowledge in color semantics and personal preferences. 

2.6. Color Semantics in Applied Arts 

Krippendorff states that “Design is making sense (of things)” [Krippendorff 1989]. This sense mak¬ 
ing in applied arts is often equated with communication via elements of design. Professional de¬ 
signers and researchers have specifically emphasized the role of colors and color semantics as a 
means of visual language [Green-Armytage 2006; Hutchings 1995]. In architecture [Hogg et al. 
1979; Caivano 2006; Ural and Yilmazer 2010], interior design [Poldma 2009], textile design [Lee 
and Lee 2006], and product design in marketing [Madden et al. 2000], understanding values of col¬ 
ors and color semantics is acknowledged as a knowledge that can support not only non-designers, 
but also professional designers who conceptualize and ideate designs based on their own intuitions. 
This knowledge can enhance the processes involved in design, such as inspiring, brainstorming, 
exemplifying, and communicating among designers over design prototypes (see [Poldma 2009]). 

2.7. Color Semantics in HCI 

Although some studies in HCI investigate the role of colors in aesthetics and usability [Lavie and 
Tractinsky 2004; Hassenzahl 2004; Hoffmann and Krauss 2004; De Angeli et al. 2006; Schmidt 
et al. 2009; Navarre et al. 2009; Moshagen and Thielsch 2010; Lee and Koubek 2010; Reinecke 
and Bernstein 2011; Sauer and Sonderegger 2011; Reinecke et al. 2013; Thuring and Mahlke 2007; 
Lindgaard 2007; Hassenzahl and Monk 2010; Lindgaard et al. 2011; Lee and Koubek 2011; Yang 
et al. 2012] and performance of the user [Moshagen et al. 2009; Sonderegger and Sauer 2010; Lin 
et al. 2013a], the notion of color semantics has been barely addressed. As an example, consider a 
scenario in which users are surfing the Web for a “massage therapy” website. Users perhaps expect 
to see a website with color combinations that impress them with “calm”, “soothing”, and “elegant” 
moods. If the color design of such a website fails to convey these moods, it will then negatively 
affect the trust, credibility, and marketing of its business (See [Lindgaard et al. 2011; Pengnate and 
Antonenko 2013]). 

We believe that color semantics is a legitimate challenge to be addressed in various venues of HCI: 
in automatic design [Lok and Feiner 2001; Gajos and Weld 2004; Hurst et al. 2009; Jahanian et al. 
2012; Hunter et al. 2011; Kuhna et al. 2012], design by example [Hartmann et al. 2008b; Herring 
et al. 2009; Lee et al. 2010; Kumar et al. 2011], design grammar [Talton et al. 2012], design of 
user interaction [Carroll 1997; Fallman 2003; Lowgren 2007; Zimmerman et al. 2007; Forlizzi et al. 
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2008; Stolterman 2008; Hashim et al. 2009; Chen 2011], quantifying aesthetics of design [Lavie and 
Tractinsky 2004; Datta et al. 2006; Reinecke et al. 2013], user experience design [Hassenzahl and 
Tractinsky 2006; Mahlke 2008; Hartmann et al. 2008a; Lowgren 2009; Law 2011; Bargas-Avila and 
Hombaek 2011; van Schaik et al. 2012], and color design [Luo 2006; Tokumaru et al. 2002; Tsai and 
Chou 2007; Wang et al. 2008; Ou et al. 2009; Hsiao et al. 2013; Jahanian et al. 2013a]. As Fallman 
argues, HCI needs to be understood and acknowledged as a design-oriented process in terms of 
philosophy and theory and methodological foundations [Fallman 2003]. Dearden and Finlay suggest 
that we need to identify patterns that are both timeless and culturally sensitive [Dearden and Finlay 
2006]. Hoffmann and Krauss [Hoffmann and Krauss 2004] conclude that many researchers do not 
recognize the importance of visual aesthetics in the communication intent. We need to work on 
semantic color vocabularies and define and propose them as domain-specific knowledge to HCI. 
This motivates us to revisit the concept of color semantics, utilize it in design mining and data- 
driven approaches to learn from the work of professional designers, and investigate theoretical and 
practical aspects of color semantics in user interaction design. 

3. RELATED WORK 

Our work lies at the intersection of four research areas: color semantics and meaning, probabilistic 
topic models, user study via crowdsourcing, and click modeling. 

3.1. Color Semantics and Meaning 

Semantics in computer science and engineering has been observed as a gap [Liu et al. 2007; Eakins 
et al. 1999; Sethi et al. 2001; Mojsilovic and Rogowitz 2001; Zhou and Huang 2000; Smeulders et al. 
2000; Chen et al. 2003] that merits more investigation. Nevertheless, in terms of color emotions and 
moods, there is a recent body of work. Csurka et al. [Csurka et al. 2010] collected a dataset of color 
combinations and their associated labels, and applied a Gaussian mixture model to the data based on 
some low level color features of the color combinations. Unlike our work, their dataset is collected 
from a limited number of good designs suggested in [Eisemann 2000] and mainly from an online 
dataset of colors [ColourLovers 2014] produced mostly by amateurs. The noisiness of the labels in 
this online dataset was acknowledged by both these authors and [O’Donovan et al. 2011]. Another 
difference is that their model does not categorize the color combinations as joint combinations of 
labels and colors. 

Applications of color semantics have recently come to the attention of researchers. In image re¬ 
trieval, Solli and Lenz [Solli and Lenz 2010] defined a mathematical framework for Kobayashi’s 
Color Image Scale. This framework is utilized by [Jahanian et al. 2013b; 2013a] in designing alter¬ 
native and customized magazine covers for non-designers based on color moods. For color mood 
transfer of images, Murray et al. [Murray et al. 2012] applied the 15 moods suggested by Csurka 
et al. [Csurka et al. 2010]. In contrast to our work, where we use human judgment to evaluate the 
goodness of our model, they use human input to build their models. In design mining the Web, Ku¬ 
mar et al. [Kumar et al. 2013] reported using expressive colors (color moods) to support data-driven 
design tools. 

In researching linguistic categorization of colors, color naming and its cross-cultural aspects are 
topics that have been under investigation for decades [Palmer 1999]. There is a body of work in 
computer science and engineering to model color naming (e.g. [Mojsilovic 2005].) In recent work, 
Heer and Stone [Heer and Stone 2012] reviewed statistical color naming models, with the goal of 
fitting a model to single colors and their associated names. These associations are either from human 
judgments or retrieved from Internet search engines. For the latter, topic modeling was explored by 
Weijer et al. [Van De Weijer et al. 2009] who used Probabilistic Latent Semantic Analysis (PLSA), 
and Schauerte and Stiefelhagen [Schauerte and Stiefelhagen 2012] who used Latent Dirichlet Allo¬ 
cation (LDA). In contrast to color naming, our color semantics work takes into consideration design 
examples, color combinations, and different levels of abstraction. Also, unlike prior work, we adapt 
topic models to jointly link colors and words, and then verify the associations through user studies. 
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3.2. LDA Topic Modeling 

The goal in LDA topic modeling is to infer underlying themes or topics of textual document cor¬ 
pora, where each topic is a multinomial distribution over words, and each document is a mixture of 
topics [Blei et al. 2003; Blei 2012]. Because for color semantics we need to infer compound (color- 
word) topics, we adapt LDA-dual [Shu et al. 2009] by viewing each magazine cover as a combined 
bag of words and colors. The closest work to our research is perhaps [Feng and Lapata 2010], where 
LDA is used for image annotation. The authors consider a document as a mixture of low level image 
features (extracted by SIFT techniques) and words. However, in our adaptation of LDA-dual model, 
we incorporate two independent multinomial distributions, one each for observed colors and words. 

3.3. User Studies via Crowdsourcing 

Viability of crowdsourcing graphical perception, including color visualization, has been confirmed 
by recent work of Heer and Bostock [Heer and Bostock 2010]. In several tests, they replicated the 
results of prior laboratory experiments using crowdsourcing and reported consistency between the 
results. The scalability yet inexpensiveness of online experiments involving color overrides some 
of their limitations, such as the inability to control for variations in visual acuity. For instance, Lin 
et al. [Lin et al. 2013a] employ crowdsourcing to confirm improvement in subjects’ performance 
when color semantics is utilized in information visualization. Reineke et al. [Reinecke et al. 2013; 
Reinecke and Gajos 2014] utilize cross-cultural crowdsourcing to study some perceived aesthetics 
aspects of visual complexity and colorfulness in a dataset of website designs. Inspired by their on¬ 
line setup, we also captured demographic information from the subjects during our crowdsourcing 
experiments. However, we emphasize that our goals from the crowdsourcing experiments are com¬ 
pletely different from work of Reinecke et al. While they wish to infer bias in perception based on 
demography and cultures, we use this information to validate the output of a statistical model. 

3.4. Click Modeling 

The goal in click modeling is to model the user’s interactions with sponsored search results or ads. 
In a recent work, Govindaraj et al. [Govindaraj et al. 2014] reviewed click models, and suggested 
a new model by taking into account relations between the user’s clicks on a list of URLs (provided 
by a search engine in response to the user’s query). Modeling the user’s back and forth clicks on 
a list of URLs, they then infer the probability of clicking on different vertical positions of URLs, 
regardless of their contents. Inspired by this work, we treat the participants responses as a click 
modeling problem. Arguably our model is simpler than that proposed by Govindaraj et al. However, 
unlike web-search where the user can choose not to click on any of the results, in our case the user 
needs to make a choice. Furthermore, since our users participate willingly in our study, the level of 
user engagement is high. Therefore, we believe that our simple model suffices. 

4. DATA COLLECTION 

Our dataset of magazine covers includes 2,654 covers from 71 magazine titles and 12 genres. We 
collected approximately 1,500 of these covers by scanning them from magazines held by a number 
of libraries and newsstands in our university. The rest of the cover images were downloaded from 
the Internet. Although we developed a web crawler tool to collect magazine covers, because many 
magazine publishers do not provide archives with high quality images, in half of the cases we had to 
collect online images by hand^. We attempted to collect roughly 12 different genres of magazines to 
capture different contexts of design. These genres include Art, Business, Education, Entertainment, 
Eamily, Eashion, Health, Nature, Politics, Science, Sports, and Technology. To this end, we loosely 
followed the Dewey Classification method [OCLC 2014a], the WorldCat indexing system [OCLC 
2014b], suggestions from our librarians, as well as the description of the magazine by the publishers. 
Table I in the appendix contains a summary of our dataset. Note that the genres are tentative. In fact. 


^This data will be made available for academic research upon request. 
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this supports the use of LDA: magazines rarely include only one topic and usually are a combination 
of different topics (or genres in this context). 

4.1. Preprocessing 

The preprocessing of cover images was performed using the Matlab Image Processing toolbox^. 
We use 512 basic colors obtained by quantizing the sRGB color space with 8 bins in each channel. 
Given this color basis, each magazine cover (image) is then a histogram of these colors. To feed the 
images to LDA-dual, we scale them to 300x200 pixels using bicubic interpolation. The down-sizing 
was done to reduce the computation without affecting the distribution of the colors in the images. 

To extract color palettes from the images, we used the color theme extraction code by [Lin and 
Hanrahan 2013]. In their implementation, the algorithm needs to take the saliency map of the given 
image (using the code by [Judd et al. 2009]), as well as the segmentation of the given image (using 
the code by [Felzenszwalb and Huttenlocher 2004]). In our work, we however used the saliency 
map code by [Harel et al. 2007], since it was easily accessible. Note that these color palettes are not 
the input to the model, but are later used to visualize the inferred color combinations. 

4.2. Word Vocabulary 

To capture the words to be associated with color distributions of the magazine covers, the words 
on the covers were transcribed by hand. To create a word vocabulary, we first prune the transcribed 
words and then create a histogram of words. Because a more meaningful vocabulary results in more 
meaningful topics, we filter out special characters, numbers, common stop words^ (e.g. articles and 
lexical words), and an additional handcrafted list of stop words (see Table II in the appendix). Com¬ 
pound words formed with a hyphen or dash are decomposed. A version of the Porter Stemming 
algorithm [Porter 1980] is used to equate different forms of a word, for instance “elegant” and “el¬ 
egance.” Finally, a mapping from month to season is applied. In order to include the context and 
classes of the magazines with the associated words, the periodical category to which each mag¬ 
azine title belongs was added to the set of words. We collected these periodical categories from 
the WorldCat indexing system, which is the largest international network of library content and 
services [OCLC 2014b]. 

5. STATISTICAL MODEL 

When ideating about visual design, the designer takes into account the topic or the context within 
which he or she is asked to convey his or her message. For instance, when the context is politics, the 
designer may tend to use darker, “heavier” and more “formal” colors. However this is not the only 
factor, the words in the design also influence the designer’s choice of colors. Figure 2 illustrates 
that pink — which is usually used for topics related to feminity — has been used in a variety of 
magazines from different genres. This observation suggests that each design’s theme might be a 
combination of words and color distributions; and each design may include a proportion of various 
themes. Our goal is to model these combinations of words and colors, and infer proportions of these 
combinations in magazine cover designs. A similar intuition has been argued in statistical topic 
modeling, specifically LDA [Blei et al. 2003], for modeling word distributions in documents as 
proportions of different word topics. 

LDA (Latent Dirichlet Allocation) is an intuitive approach to infer topics from text data. As 
Blei et al. [Blei et al. 2003; Blei 2012] describe, instead of categorizing and exploring documents 
using tools such as keywords, we may first categorize documents based on topics. This allows us to 
explore topics of interest and find related documents. For example, a document about sociology may 
include different topics, such as biology, evolution, history, and statistics with different proportions. 
Each of these individual topics can be viewed as a multinomial distribution over a fixed vocabulary 
of words. Accordingly, each document, which can be viewed as a bag of words, is a combination of 


^The MathWorks, Inc., Natick, MA. 

^Provided by MySQL database, available at https://dev.mysqLcom/doc/refman/5.1/en/fulltext-stopwords.html. 
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Fig. 2: Pink is used in all of these designs, despite the fact that each of these designs belongs to a 
different context and genre of magazines. 


these topics with some proportions. Typically, a value for the number of topics is chosen by hand. 
The latent topics, as well as the topic proportions of each document, are inferred by LDA using the 
observed data, which are the words in the documents. 

Just as word topics are distributions over words, one may think of color topics as distributions 
over colors. One may argue that by only using LDA, we might be able to model the distribution 
of the colors on the covers, and infer the color topics. We could then utilize crowdsourcing, and 
ask participants to label the inferred color topics to include combined color-word topics, and hence 
the color semantics associations. The problem with this approach is that we cannot capture the 
associations created in each design by the designer. Moreover, we cannot perform this modeling in 
an automatic fashion, since we need human judgements to label the color topics, and this labeling 
will be performed by naive subjects. Hence, we will lose the valuable insight that the designer put 
into selecting color and word topic combinations. 

A better approach is to model the links or the associations between the color topics and word 
topics to infer combined color-word topics. This means that the LDA model framework needs to 
be extended. Such an extension was recently proposed by Shu et al. [Shu et al. 2009]. The key to 
this extension is the proportions vector 6. In our case, for each cover design, the proportion is a 
combination of color assignments, as well as word assignments. Similar to LDA, these assignments 
are modeled with multinomial (and conjugated by Dirichlet) distributions. 

5.1. Review of LDA-dual Model for Color Semantics 

In this section, we review the LDA-dual model proposed by Shu et. al [Shu et al. 2009], and explain 
how to adapt this model for color semantics. Our implementation^ is an extension of the Matlab 
Topic Modeling toolbox [Steyvers and Griffiths 2014] developed by [Griffiths and Steyvers 2004] 
for LDA. 

Assume that there are K color-word topics denoted by , /c 2 ,..., and D magazine covers 
denoted by di, ^ 2 ,d^:). Let W denote the number of words in the vocabulary and C denote the 
number of color swatches, where each swatch is a patch of color defined by using its sRGB values^. 
Moreover, let denote the number of words and denote the number of color swatches in 
magazine cover da. Let Wd,m denote the m-th word in the d-th document and Cd^n denote the n-th 
color swatch in the d-th document. Each magazine cover includes some proportion of each word 
topic, as well as each color topic. Let yd^m denote the word topic assignment to the word Wd^m 
and Zd,n denote the color topic assignment to the color swatch Cd^n- Note that these assignments 
are latent. Also let and denote the multinomial distributions of the word topics and the 
color topics, respectively. 


"^Our implementation is available upon request. 

^Recall that we discretize and use eight values for each of the three sRGB color channels. Therefore, C = 512. 
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Each magazine cover includes some proportion of the color-word topics. These proportions are 
latent, and one may use the K dimensional probability vector Od to denote the corresponding multi¬ 
nomial distribution for a document dd. 

Let (3, 7 , and a be the hyper-parameters of the three Dirichlet distributions for the color topics, 
word topics, and the proportions Od, respectively. Let Dirichlet(-) denote the Dirichlet distribu¬ 
tion, and Discrete(Dirichlet(-)) denote the discrete distribution that is drawn from a Dirichlet 
distribution. 

Given the above notation, the generative model for LDA-dual can be written as follows: 

(1) Draw K word topics ^ Dirichlet (7). 

(2) Draw K color topics (j)k ^ Dirichlet(;5). 

(3) Lor each document dd G {di, (i 2 ,... ^dn}' 

— Draw Od ^ Dirichlet (a). 

— Lor each word Wd,m with m = 1,..., Md 
— Draw yd,m ^ Discrete(6>rf) 

— Draw Wd,m ^ Discrete('0^^ 

— Lor each color Cd^n with n = ... ^Nd 

— Draw Zd^n ^ Discrete(6>c^) 

— Draw Cd,n ^ Discrete(0^^ 

A graphical model for this generative process is illustrated in Lig. 3, where the shaded nodes 
denote observed random variables and the unshaded nodes are latent random variables. 



Lig. 3: Hierarchical Bayesian plate model for the LDA-dual model, which combines color and word 
topics. Here, D is the number of magazine covers; K is the number of color-word topics; and Nd 
and Md are the number of color swatches and words, respectively, in the d-th magazine cover. 


If we let <p = {(j)i,...,(t)K}, = {ipi,...,ipK}, 0 = {01,..., 0d}, Zd = {zd,i,... ,Zd,NA, 

Vd = {yd,i,---,yd,Md}, Z = {zi,...,zd}, y = {yi,---,yD}, W = {wi,...,Wd}, and c = 
{ci,..., Cd}, then the joint distribution corresponding to the LDA-dual model above can be written 
as 

K D / N 

p{4),'4>,d,Z,C,y,w) = ' ( '[[p^^d,n\0d)p{cd,n\(f>,zd,n) 

i=l d=l \n=l 

M \ 

n p(yd \0d)p{wd ,m 1'0,2/d ) • (1) 

m=l / 

Ligure 4 provides a graphical illustration of the generative mechanism and the inference proce¬ 
dure described below. This figure is a symbolic representation of the model. In each sub-figure, a 
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cylinder represents a color-word topic. Each arrow represents the probability of each cover being 
drawn from a given color-word topic. Each cover includes a histogram of colors and a list of words 
(each word is superscripted by its corresponding color-word topic). In the generative process, we 
know the distribution of the color-word topics, and can produce the distribution of the colors and 
words on the magazine covers. Eor instance, “Cover 1” is completely (with probability 1.0) gener¬ 
ated by color-word topic 1. “Cover 2” is generated by equal distributions of both “color-word topic 
1” and “color-word topic 2”. In the statistical inference mechanism, we only know the distribution 
of the colors and the words for each cover. We do not know (represented by question marks) the 
color-word topics, their proportions, and the assignments of the colors and words of each cover to 
these color-word topics. 


PROBABILISTIC GENERATIVE PROCESS 


STATISTICAL INFERENCE 



Color-word topic 1 




beauty^ women^ 
fashion^ beauty^ 



beauty^ fashion^, 
art^, art^, fashion^ 


^asb'on 




Color-word topic 2 



house^, fashion^, 
art^, house^, fashion^ 




Color-word topic 1 


Color-word topic 2 



beauty’, women’, 
fashion’, beauty- 



house’, fashion’, 
art’, house’, fashion’ 


Eig. 4: EDA is both a generative and inference model. This image is inspired by [Steyvers and 
Griffiths 2007]. 


5.2. Inference 

Since c and w are observed, inference entails computing 




p{(f),ip,6,z,y,c,w) 

p{c,w) 


( 2 ) 


Theoretically, the above distribution can be obtained by computing the joint probability distri¬ 
bution of the latent and the observed variables, and then computing the marginal probability of 
the observations. In practice, however, topic modeling algorithms approximate the result to bypass 
the computational complexity of the solution. There are often two approaches for this approxima¬ 
tion [Blei 2012]: variational inference [Jordan et al. 1999; Teh et al. 2006] and Markov chain Monte 
Carlo (MCMC) sampling [Andrieu et al. 2003; Griffiths 2002]. Eor the sampling approach, the col¬ 
lapsed Gibbs version of sampling is discussed in [Porteous et al. 2008; Griffiths and Steyvers 2004; 
Shu et al. 2009]. We extend MCMC collapsed Gibbs sampling version implemented in the Matlab 
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Fig. 5: Color-word topics inferred by the LDA-dual model, (a) Illustration of the 12 color topics 
in the middle, and their corresponding 12 word topics; 6 on top for the first 6 color histograms 
from left, and the other 6 on the bottom. Note that for visualization, only the principal elements in 
the histograms are shown. Also note that the numerical weight of each word topic is shown next 
to heading of each word topic histogram. Similarly, the numerical weights of the words in each 
word topic histogram are shown next to their corresponding words (see Sec. 5.2). As examples, 
the proportions of the 12 color-word topics in (b) Horticulture, (c) Time, and (d) Vogue magazines 
(including all the issues in the dataset from year 2000 to 2013) are shown. Note that /c’s are the same 
as in (a). See Fig. 14 for all of the magazines. 
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Topic Modeling toolbox [Steyvers and Griffiths 2014] developed by [Griffiths and Steyvers 2004] 
for LDA (see [Jahanian 2014] for our derivations of LDA-dual). 

Although some variants of LDA can automatically find an optimal value for the number of topics 
AT, based on our domain knowledge from the data collection process, we simply set AT = 12. 
Because each color-word topic includes proportions of the color basis and the vocabulary words, 
we visualize a topic as a pair of colors and words histograms. Figure 5 (a) illustrates the 12 topic 
histogram pairs for K = 12, a = 0.8, and = 7 = 0.1. We tuned the values for a, p, and 7 
by trial and error to produce visually pleasing color histograms and semantically meaningful word 
histograms. The visualized histograms just illustrate the principal components. Of course, other 
levels of granularity can be visualized if needed (see [Jahanian 2014] for more examples). Note 
that in this figure, similar to “WORD_TOPIC_l”, “COLOR_TOPIC_l” is a distribution over some 
colors. WORD_TOPIC_l itself has some proportion in the entire dataset (0.0483). The summation 
of all the 12 word topics proportions is 1. Similarly, all the 12 color topic proportions add up to 1. 
Here, just for visualization, we show the length of each color topic based on its ratio to the color 
topic —COLOR_TOPIC_4, which has the largest proportion. 

Figures 5 (b), (c), and (d) illustrate the proportions of each of the inferred color-word topics for 
three magazine title designs in the dataset. For instance, note that Vogue as a fashion magazine has 

and /cg as two of the dominant color-word topics. As can be seen, and /cg contain words such as 
“women”, “fashion”, “love”, and “beauty”, while the corresponding color histograms contain pastel 
and pink colors, which are often associated with fashion magazines. On the other hand. Horticulture, 
which is a nature magazine, has the highest proportion of ki, which pre-dominantly contains shades 
of green. The words in ki include gardening-related words such as “gardens”, “landscapes”, and 
“plants”. See Fig. 14 for all 71 magazine titles. 

6. INTERPRETING THE MODEL OUTPUT 

Visualizing the results of LDA is a topic of research [Chaney and Blei 2012; Chuang et al. 2012]. 
Chaney and Blei [Chaney and Blei 2012], for example, suggest a visualization mechanism for ex¬ 
ploring and navigating through inferred topics from LDA and their corresponding documents. Al¬ 
though their work does not completely address the usability evaluation of this mechanism, it inspired 
our visualization mechanism for our user study. In order to evaluate the color semantics hypothesis, 
we need to display both the color histogram and the word histogram to the participants in our user 
study in a comprehensive, yet unbiased fashion. We address this via a two-step process. The word 
histograms are converted to word clouds, while the color histograms are converted to color palettes 
using the mechanism described below. Figure 6 illustrates the visualization process. 

6.1. From Color Histograms to Color Palettes 

We use 5-color palettes as proxies to represent each of the color histograms in Fig. 5 (a). These 
palettes are drawn from a pool of 5-color palettes, one for each magazine cover in the dataset (refer 
to Sec. 4.1 for the mechanism of creation of this pool). 

The intuition behind using 5-color palettes is that when designers are asked to design a media 
piece, they often choose a 3-color, 5-color, or occasionally a 7-color palette, and consistently use it; 
so that their designs are clean and sophisticated, as opposed to busy and cluttered. 

In order to find the 5-color palettes that are closest to the color topic histograms, we define a sim¬ 
ilarity metric as follows: Let denote a color topic histogram with the 512 color basis defined 
earlier, and denote a 5-color palette. An intuitive similarity metric is the Euclidean distance be¬ 
tween color swatches of and . Among the possible color spaces, we choose the CIE L*a*b* 
color space with a D65 reference white point. It is considered to be a perceptually uniform space, 
where A A around 2.3 (the distance between two colors) corresponds to one JND (Just Noticeable 
Difference) [Sharma 2002]. 

Defining the color similarity distance problem as a bipartite graph matching between and 
with 512 and 5 nodes, respectively, we find the minimum distance cost of this graph using 
the Hungarian method [Kuhn 1955]. Equation 3 defines the weighted Euclidean distances dwED 
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Fig. 6: Visualization process for the inferred color-word topics. To visualize the color-word topic 
histograms inferred by the model (see Fig. 5), we use 5-color palettes and word clouds as proxies 
to color histograms and word histograms, respectively. *See Sec. 4.1 for the color theme extraction 
details. 


between the nodes of these two graphs. Here, the weight Wi corresponds to the weight of the i- 
th color in the color topic histogram and ~ II 2 ^^^^tes the distance in CIE L*a*b* 

between the i-th color from and the j-th color from . This metric can be thought as a version 
of The Earth Mover’s distance suggested by Rubner et al. [Rubner et al. 2000] for image retrieval, 
with the weight vector representing color importance. 


512 ^ 5 

dwED = / , - / , 

z—^ 'in- z—rf' 


- s: 


.Wi 
1=1 J=1 


3 II 2 ' 


(3) 


Computing dwED for a given color topic histogram and all 5-color palettes, we can choose the 
closest of them as proxies to the histogram (see Eig. 6). In the user study, we present two series of 
questions for the first and the second closest color palettes, because just one color palette may not 
provide an adequate visualization of the entire topic histogram. See Eig. 13 for the entire color-word 
topics. 


6.2. From Weighted Bag of Words to Word Clouds 

Eigures 6 illustrates how we visualize each word topic histogram with a word cloud. The word 
cloud (or tag cloud) is a visualization technique used to show the relative weights of words through 
different font sizes. The weights resemble frequency of occurrence or importance of the words in a 
word dataset. A suite of word cloud algorithms and their usabilities are discussed in [Seifert et al. 
2008]. Because of the popularity of word clouds in visualizing categories, and the fact that words 
are randomly scattered over a layout, we used this technique in our user study. Using wordle^, we 
generated black and white word clouds to avoid introducing any color bias. Note that whereas we 


® http: //w w w. wordle .net 
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chose two color palettes for each color topic, we developed only one word cloud for each word 
topic. 

7. USER STUDY 

The main aim of this section is to validate the output of the probabilistic topic model. In particular, 
we want to understand if casual users (who are not necessarily designers) agree with the association 
between color combinations and linguistic concepts produced by our model. 

7.1. Participants 

Our color semantics online survey^ is hosted at our university’s survey system. It was advertised 
through social networks and the university email network. So far, we have collected 859 responses 
from 487 (56.69%) females, 367 (42.72%) males, and 5 others (0.58%), in the age range of 18 
to 80 years (with mean = 30.98). The participants are from 70 countries and natively speak 66 
different languages, with the majority from the US (59.84%). There are 348 (40.51%) participants 
who have lived in more than one country. There are 352 (40.97%) participants with college degrees, 
451 (52.50%) with graduate degrees, and 55 others (pre-high school, high school, and professional 
degree). The majority of the participants, 716 (83.35%) are non-designers. In contrast, there are 130 
(15.13%) participants with three or more years of experience in visual design (including graphic 
design, interior design, and textiles.) 


Demography 

questions 



Q: From your opinion, 
which of these word 
clouds represent this 
color palette? Choose 
as many as apply. 



Another 

set? 


S \ ^ I 

Yes ' \ \ /^ 

□ scieno^f^! - ) K ^ 

y ^8 times 

No DSTf^i-dins^.^ [ 

CH None of the above | 


^ Randomly selected Shuffled 



Fig. 7: Flow of the user study. 


7.2. Stimuli and Procedure 

Figure 7 illustrates the flow of the survey. In order to simulate a matching experiment between pairs 
of color and word topics, we designed a question as follows: one 5-color palette is shown in the left 
side of the screen, and three shuffled and randomly chosen word clouds, as well as a “None of the 
above” option are shown on the right side arranged in vertical order. Each question is a multiple 
choice (represented by multiple choice check boxes). For each question, we asked the participant 
to choose as many word clouds, as in his/her opinion apply to the 5-color palette shown. If the 
participant felt that none of the word clouds applied to the 5-color palette, he/she could choose 
“None of the above”. Among the three randomly drawn word clouds, one is the word cloud inferred 
from the model. 


'^http://goo.gl/P4W9XL 
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The survey is divided into two subsets of questions. The reason for this is that otherwise some 
participants may lose interest in finishing the experiment. This observation was made in the pilot 
experiment More specifically, we created 24 questions for the first and second closest 5-color 
palettes corresponding to the 12 inferred color topics. However to avoid exhausting the participants, 
we randomly draw 8 questions from the 12 questions of the closest color palettes and ask the partic¬ 
ipants to answer them. Then we ask the participants if they would like to continue by taking another 
set of 8 questions (this time drawn from the 12 questions of the second closest color palettes). Of 
all participants, 61.35% of the users chose to continue, and answered all 16 questions. 

Note that the reason for shuffling the word clouds in each question is because we conjectured that 
there might be a position bias in the vertically arranged options. In Sec. 8, we explain how we were 
inspired by the click model in [Govindaraj et al. 2014], and utilized this framework to analyze the 
experimental results. 

8. INTERPRETING THE USER STUDY 

In this section we explain the statistical inference mechanism that we used to understand the user 
responses. 

8.1. Statistical Model 

First, we define some notation. Let Ci denote the event that the i-th color palette was displayed. 
Also, let Wj denote the event that the user selected (clicked on) the j-th word cloud, and Uij denote 
the probability that the j-th word cloud was selected by the user in response to the i-th color palette. 
In order to compute Uij we note that 

Uij =Pr{wj\ci). (4) 

There are three possible positions p € {1,2,3} at which a word cloud can he displayed. Let djp 
denote the event that the j-th word cloud was displayed at position p, and let Wjp denote the event 
that the user selected the j-th word cloud which was displayed at the p-th position. Then 

Uij = ^ ^ Pr(rCjfp q) ’ PY(^djp\ci^. (5) 

pe{i,2,3} 

If dj denotes the event that the j-th word cloud was selected for display and d^ the event that it was 
displayed at position p, then 

Pr{djp\ci) = Pr{dj\ci) ■ Pr(d^|ci). (6) 

According to our experimental design, each word cloud has an equal probability of appearing in any 
one of the three positions. Therefore 

Pr(d^|ci) = (7) 

On the other hand, we always select the i-th word cloud (the true word cloud according to our 
model) for the i-th color palette. The other two slots are filled by selecting any two of the remaining 
11 word clouds uniformly at random. Therefore 

Pr{dj\ci) = l\ ( 8 ) 

^ ^ otherwise. 

All that remains is to estimate FY{wjp\djp, q). For this task, we borrow from the cascade click 
model [Govindaraj et al. 2014] and write the probability as a product of the following two factors: 

— the probability that the p-th position is examined by an user, the so-called position bias. It is 
denoted as bp. 


®This pilot experiment is hosted at http://goo.glAVFnjHL. The data collected for this experiment is available upon request. 
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— the intrinsic relevance of the word cloud j to the color palette i. This is the quantity that we seek 
to infer from the user responses; and we will denote it as Vij. 

In other words, 


Vv{wjp\djp^Ci) — Tij ' bp^ 

and by using (5) and letting qij = 2 3 } ' bp, we can write 

X] ^r{djp\ci) ■ bp = Tij ■ Qij. 
pe{i,2,3} 

Note that bp can be pre-computed as follows: 


(9) 

( 10 ) 


( 11 ) 


where m denotes the total number of trials (each question in our survey is equivalent to one trial), 
and rrip denotes the number of times the word cloud at position p was selected in any of the trials. 
In our experiments, we found the position bias of the options (in the vertical order) for the first set 
of the questions to be 0.323, 0.3626, 0.3381,0.1499 (in turn); and 0.4089,0.3884, 0.351, 0.1364 (in 
turn) for the second set of the questions. Note that for each set, these numbers do not sum up to 1, 
because of the fact that the participant could choose more than one word cloud. 

These numbers indicate that the position bias for each option is not equal, and even though we 
shuffled the three choices of word clouds in the first three vertical positions, we need to account for 
the position bias. We note that the fourth option —“None of the above”— is clicked less than the 
other options. This indicates that our participants wished to provide an answer, as well as the fact 
that they may have not thought that the associations between the word clouds and the colors were 
too abstract. We also note that in the second set of the questions, the fourth number is lower than the 
one in the first set of questions. This perhaps means that the participants who chose to participate in 
one more set of the questions in the survey were more confident regarding their conclusions. 

As the last step, let rrii denote the number of trials in which the i-th color palette was displayed, 
and rriij denote the number of trials in which the i-th color palette was displayed and the j-th word 
cloud was selected. We can assume that the trials are independent, and therefore the probability of 
observing this data under model (10) can be written as 


Pr(mi, rriij) = {rij • (1 - . (12) 

The maximum likelihood estimate for Vij • qij is simply from which we can infer Vij, the 
maximum likelihood estimate for Vij , as 


rrii 


rrii • qij 


(13) 


8.2. Analyzing the Results 

Figure 8 illustrates two relevance matrices and (resp. corresponds to the inferred rel¬ 

evance of the first (resp. second) closest color palette to the word cloud produced by LDA-dual. The 
rows are color palettes, as proxies to color topics histograms, and the columns are word clouds, as 
proxies to word topics histograms. The (i, j)-th elements of these matrices are the intrinsic relevance 
values Tij , computed from the observed responses of the participants using the model described in 
the previous section. Higher values of Vij mean that the users found a high correlation between the 
i-th word cloud and the j-th color palette. If the participants find the word cloud produced by our 
model to be the most relevant for a given color palette, then the diagonal entries, marked in blue, 
should contain the highest values. Whenever an off-diagonal entry is larger than the corresponding 
diagonal entry, it is marked yellow in the figure. 
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Fig. 8: Relevance matrices and for the first and second set of questions, respectively. For the 
first set of questions, the participants were shown the closest palettes identified by LDA-dual. For 
the second set of questions, the subjects were shown the second closest palettes identified by LDA- 
dual. The elements of these matrices are the estimated intrinsic relevance of associations between 
colors and words, calculated from the participants’ responses. The higher the value, the greater the 
intrinsic relevance associated by the users. Ideally, the diagonals should contain the highest values. 


Note that for the first set of 12 color palettes, subjects selected the “None of the above” option on 
average for 13.42% of the trials. The minimum was 7%, which occurred for ki \ and the maximum 
was 28%, which occurred for k 2 . For the second set of 12 color palettes, the average was 12.83%. 
The minimum was 2%, which occurred for ki; and the maximum was 22%, which occurred for 
k 2 . Frequent selection of the “None of the above” option for a given color palette suggests that 
participants had more difficulty associating this palette with the word clouds that were shown. In 
the relevance matrices, we do not compute the numbers for each color palette against the “None of 
the above” option. Thus the matrices do not contain the 13-th column. However, if the participant 
has selected “None of the above” as well as other options, we take into account those options. One 
reason for this decision is when the participant chooses an option along with this option, he/she may 
feel that the other option applies but he/she is not certain about it. 

As it is observed from the relevance matrices in Fig. 8, in most cases the diagonal elements are 
higher than the off-diagonal elements. This indicates a strong correlation between the results of 
LDA-dual and the opinion of the participants (also see the next section for an aggregated measure). 
We studied the entries with the highest relevance values such as rh and rgg. Referring to Figs 13 

(a) and (i), one reason these values are high could be because green for “garden” (rj^) and pink 
and purple for “fashion” and “women” (fgg) are intuitive and widely accepted associations across 
different cultures. 

There are a few color palettes such as C 2 in R^ where the users assign higher relevance to word 
clouds other than the one produced by the LDA-dual model. To understand this, note that in Fig. 13 

(b) , the first 5-color palette which is C 2 predominantly contains shades of red and black. Unsurpris¬ 
ingly, users assign higher relevance to word clouds wg and wg, which are about “sex” and “beauty.” 
In our dataset however, the red and black color combinations are often used by news magazines such 
as Time and The Economist, and computer magazines such as PC Magazine. One can perform a sim¬ 
ilar analysis for other color palettes such as ci 2 in R^ and cg in R^ to infer why there is a mismatch 
between the model output and the relevance values assigned by the users. This also shows why it 
is important to perform a user study; domain specific color palettes and their associated linguistic 
concepts may not always transfer to a general context. 

To understand differences between female versus male and non-US versus US participants, we 
computed the corresponding relevance matrices (see Fig. 15). Comparing these matrices with Fig. 8, 
we do not observe any significant differences. This indicates that our results do not depend signifi¬ 
cantly on the gender or cultural background of the users. However, when we compare the designers 
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versus non-designers, we note that there are more zero values in the off-diagonals for designers. 
This indicates that designers exhibit a stronger bias against selecting certain word clouds for certain 
color palettes, perhaps because of their training [Whitfield and Wiltshire 1982]. 

8.3. Aggregated Measures 

We have defined two different measures to assess the strength of the relationships of elements in a 
matrix as indicated by the magnitude of the diagonal elements relative to the off-diagonal elements 
in each row. For the definitions below, it is assumed that the matrix elements are all non-negative, 
and are defined from 


R = [vij] , (14) 

for each relevance matrix and . 

The first measure is the diagonal dominance D. For each row, it is simply the ratio of the diagonal 
element in that row to the average value of the non-diagonal elements in that row: 

A = (15) 

Ti 

where vu is a diagonal element, and r'i is the mean of off-diagonal elements: 

1 ^ 

i=i 

To get a summary assessment, we can average the diagonal dominance over all the rows of the 
matrix: 


1 ^ 
i=l 

For the two matrices R^ and R^ in Fig.8, the average diagonal dominances are 2.05 and 1.99, 
respectively (see Fig. 9). So on average, the diagonal element is twice as large as the other elements 
in the row. This suggests that despite the wide variability of the data in the matrices, the diagonal 
elements do tend to dominate. 


D S 
2.05 2.08 

1.99 1.74 

Fig. 9: Aggregated measures for relevance matrices R^ and R^. Column D is the diagonal dom¬ 
inance measure computed according to (17). Column S is the diagonal separation computed ac¬ 
cording to (20). Overall, these two separate measures support the conclusion that on average, the 
diagonal elements in the relevance matrices are significantly stronger than the off-diagonal elements. 




The second measurement is the diagonal separation S. It is also defined for each row, and is also 
a ratio: 





( 18 ) 
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The numerator is the absolute value of the difference between the diagonal element and the mean of 
the off-diagonal elements in that row. The denominator is the standard deviation of the off-diagonal 
elements in that row: 


r 




1 \ 
, i=i 

V 



(19) 


To get a summary assessment, we can again average the diagonal separation over all the rows of 
the matrix: 


5 


1 ^ 




( 20 ) 


As Fig. 9 summarizes, for matrix (Fig. 8 (a)), the average diagonal separation is 2.08. Thus, 
on average, the diagonal element is more than two standard deviations away from the mean of the 
off-diagonal elements. This is a very good level of separation. For the second matrix, (Fig. 8 
(b)), the average diagonal separation is 1.74. While this is not as large as it is for the first matrix , it 
still indicates very good separation. 

Overall, these two separate measures support the conclusion that on average, the diagonal ele¬ 
ments are, indeed, significantly stronger than the off-diagonal elements. 


9. APPLICATIONS 

Similar to the applications of color naming proposed by Heer and Stone [Heer and Stone 2012], 
in order to illustrate how color semantics enables more meaningful user interactions, we present a 
number of applications in color palette selection and design example retrieval, image retrieval, and 
recoloring images using semantics. 


9.1. Color Palettes Selection Using Semantics 

One way to make design accessible to the masses is to recommend design elements and design 
examples. In content creation, for instance, the user needs to use a color combination that is both 
appealing and aligned with his/her purpose of design. While this need seems more immediate for 
the non-designer, a designer may also prefer to see examples for a more creative or even an inspir¬ 
ing color combination. Our inferred color semantics can be applied in color palette recommenda¬ 
tion. Although there are pools of color palettes available online [Incorporated 2014; ColourLovers 
2014], it is not easy to navigate, find, or recommend a set of palettes based on the user’s need 
(see [O’Donovan et al. 2011] and [O’Donovan et al. 2014]). Consider a scenario in which the user 
wishes to find a color palette that conveys meanings of both “technology” and “fashion.” Figure 1 
illustrates such a scenario, where the user queries for “techy fashion”. This type of query is mapped 
to the word topics, and from them to their corresponding color topics. We then map these color 
topics to a set of color palettes in a pool of color palettes, created from the magazine dataset using 
techniques in [Lin and Hanrahan 2013]. The user can then choose his/her preferred color palettes 
from the recommended set. 


9.2. Design Example Recommendation 

Another plausible application that could use our color semantics is recommendation of alternative 
designs. This can be done in several ways, i.e. by using a given a set of recommended color palettes 
(see Fig. 1), or directly, just by taking color semantics as the user’s input. Providing such examples 
can be utilized in creativity support tools [Shneiderman 2009]. It can also facilitate design proto¬ 
typing [Dow et al. 2010]. Similarly, translating the associations inferred from the magazine covers 
to other kinds of media such as website remains a topic for future work. Because LDA-dual is a 
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generative model, it can be used in automatic creation of designs based on the user’s preferences, 
similar to magazine cover design [Jahanian et al. 2013b], and website design [Kumar et al. 2011]. 
Utilizing generative models can also leverage the design-driven studies in HCI [Talton et al. 2012]. 

9.3. Image Retrieval Using Color Semantics 

The gap of color semantics in computer vision, specifically for image retrieval, has been emphasized 
by prior work [Smeulders et al. 2000; Sethi et al. 2001; Mojsilovic and Rogowitz 2001; Liu et al. 
2007]. Nevertheless, the associations of color combinations and linguistic concepts can be utilized 
in designing high level image features for both current image retrieval algorithms [Solli and Lenz 
2010] as well as deep learning algorithms [Couprie et al. 2013; Socher et al. 2013]. Figure 10 
illustrates an application of our inferred color semantics in image retrieval. Consider a scenario in 
which, the user first makes a query in an image search community, e.g. Pinterest [Pinterest 2014] 
about “interior design”. As a results, a number of images will be retrieved and suggested to the 
user. However, in order to explore and navigate through the retrieved images, the user may query 
for “garden-inspired” images. In this case, we are able to map this query to the color-word topics, 
and rank the already retrieved images based on the color topic histograms that represent “garden”. 



Fig. 10: Hypothetical application of image retrieval using semantics. The retrieved images on the 
right side are ranked vertically. In this vertical order, a larger image represents a higher rank. Images 
from [Pinterest 2014]. 


9.4. Image Color Selection Using Semantics 

As Heer and Stone [Heer and Bostock 2010] note, a common interaction in image editing is to find 
a region of colors. They suggest two types of color region mechanisms based on color names: color 
name queries from the user, and using the magic wand tool. In our case, we use the user’s query and 
present the set of pixels that semantically contributes to the colors of an image. Figure 11 illustrates 
this kind of interaction. Figure 11 (a) is the original image, a screenshot of a travel agency web¬ 
site [TripAdvisor 2014] nominated as one of the best designs (based on people’s votes) according 
to the Webby Awards 2014 [WebbyAwards 2014]. We are interested to know what color regions 
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have contributed to “travel” and “shop” in this image. Figure 11 (b) represents these regions, while 
turning the other regions to grayscale mode. 


E - E 



(a) (b) 

Fig. 11: Image color selection using color semantics, (a) The original image, (b) colors that con¬ 
tribute to “travel” and “shop” in the original image. Image from the home page of Tripadvi- 
sor.com [TripAdvisor 2014]. 


Note that in order to find these color regions we map the user’s queries to the word topics, and pre¬ 
serve their associated color topics in the image. Of course, utilizing the current image segmentation 
techniques leads to a smoother pixel selection. 

9.5. Image Recoloring Using Semantics 

Given the fact that colors communicate messages, designers often modify color themes of their de¬ 
signs. Part of this color modification may involve recoloring an image [Lin et al. 2013b], transferring 
a color theme to an image [Murray et al. 2012], or enhancing a color theme of an image [Wang et al. 
2010]. In this section we suggest an application of color semantics for colorizing patterns (Fig. 12), 
based on techniques introduced by Lin et al. [Lin et al. 2013b]^. 

Figure 12 (a) illustrates a grayscale pattern. Using the color semantics associations, we are able 
to accept a user query, map it to the word topics, and use a 5-color palette as a representative of this 
color topic to colorize the original pattern. Figures 12 (b), (c), and (d) illustrate the results of col¬ 
orizing the original pattern using “science”, “shop”, and “sports” and “men” queries, respectively. 



(a) (b) (c) (d) 

Fig. 12: Pattern colorization using color semantics, (a) The original pattern, (b) “science”, (c) 
“shop”, and (d) “sports” and “men” colorized patterns. Images colorized using techniques in [Lin 
et al. 2013b]. Pattern from ColourLovers ArrayOfLilly [ColourLovers 2014]. 


®The recolored patterns in this paper are produced by S. Lin, at our request. 
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10. CONCLUSION AND FUTURE WORK 

Taking into account that the goal of visual design is both to convey a message and to appeal to 
audiences aesthetically, we investigate design mining of designers’ thought processes in associating 
colors with linguistic concepts. We collected high quality examples of good designs, a dataset of 
2,654 magazine cover designs in 71 titles and 12 genres. We then adapted LDA-dual, an extension 
of the popular LDA topic model, to simultaneously explain both designers’ choice of colors and 
choice of words on the magazine covers. To verify the inferred color semantics with the wisdom 
of the crowd, we used a crowdsourcing experiment. The results confirm that our model is able to 
successfully discover the association between colors and linguistic concepts. This completes the 
loop of our design mining system, from hypothesis inference to validation. 

Our user study shows that users across different countries, gender, and age groups largely agree 
with the colors and linguistic concepts associations produced by our model. However, to put this 
in context, note that our experiment is biased toward an audience that is capable of reading and 
understanding the English language, is mainly college educated, and has access to the Internet. 
Arguably, the Internet is globalizing design by melting cultural perceptions [Carroll 2007]. Never¬ 
theless, studying color semantics in different cultures and folklore (similar to [Reinecke and Gajos 
2014] in studying aesthetics of low level features of colors) can increase our understanding of cus¬ 
tomized designs and hopefully help both designers and non-designers in communicating with their 
audiences. 

We then suggest a number of applications for color semantics to demonstrate how semantics can 
enable more meaningful user interactions, and perhaps help masses to design better. We specifically 
discuss color palette selection and design example recommendation, image retrieval, color region 
selection in images, and pattern colonization in image recoloring. Nevertheless, at this stage, these 
applications are not final tools. They need to be investigated more in terms of user experience design, 
and to be evaluated how much they can actually facilitate user interactions. Incorporating these 
applications in current design creation tools is also part of the future work. 

The present work is a first step towards our broader goal of making design accessible to the 
general public. We believe that this work can open up a number of interesting avenues for future 
work, some of which we now discuss. 

10.1. Visual Design Language and User Interaction 

From a broader perspective, association of linguistic concepts with design elements, specifically 
colors, can accelerate the development of a visual design language. This can enhance the user in¬ 
teraction mechanisms to be more intuitive and cognitive, and the user’s involvement in design tasks 
such as prototyping, collaboration, and critique. The LDA-dual model can be utilized to associate 
the critiques (in a design critique system such as [Xu et al. 2014]) and semantic tags (in a design 
query system such as [Kumar et al. 2013]) with design elements of the corresponding designs. This 
may provide more insight to understanding design critiques and queries. 

10.2. Quantifying Aesthetics 

Current computational models for quantifying aesthetics are based on low level features of design 
elements [Datta et al. 2006; Li and Chen 2009; Joshi et al. 2011; Reinecke et al. 2013]. We believe 
that both outcomes of this work, the inferred color semantics and the application of LDA-dual 
model to color semantics, can leverage the current models of quantifying aesthetics. While the first 
outcome is intended to provide more high level features to evaluate a design based on the message 
that it is meant to convey, the second outcome can be used in linking linguistic concepts with colors, 
spatial composition (layout), and typography of visual media design. 
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APPENDIX 


Table I: Summary of Our Magazine Covers Dataset. 


Magazine Title 

# Collected 

Magazine Title 

# Collected 

Art in America 

49 

Bloomberg Businessweek 

50 

ARTNews 

50 

Entrepreneur 

52 

HOW 

41 

Eorbes 

50 

ID 

1 

Harvard Business Review 

50 

Interior Design 

50 

Money 

48 

Live Design 

9 

The Economist 

29 

The New Yorker 

50 







Art Total: 

250 

Business Total: 

279 

Magazine Title 

# Collected 

Magazine Title 

# Collected 

Academe 

40 

Conde Nast Traveler 

50 

Language 

36 

Jet 

50 

Language Arts 

2 

National Geographic 

44 

Social Studies and the Young Learner 

50 

People 

50 

Teaching Exceptional Children 

2 

Rolling Stone 

50 

Techniques 

48 



The American Scholar 

3 



The Reading Teacher 

3 



The Science Teacher 

50 







Education Total: 

234 

Entertainment Total: 

244 

Magazine Title 

# Collected 

Magazine Title 

# Collected 

Good Housekeeping 

61 

Esquire 

11 

Better Homes and Gardens 

30 

Essence 

50 

House Beautiful 

50 

Glamour 

50 

Parenting 

51 

GQ 

50 

ShopSmart 

40 

Vogue 

50 





Eamily Total: 

232 

Eashion Total: 

211 

Magazine Title 

# Collected 

Magazine Title 

# Collected 

Health 

17 

BBC Wildlife 

25 

Men’s Health 

50 

Garden Design 

35 

Natural Health 

17 

House & Garden 

15 

Women’s Health 

50 

Organic Gardening 

17 





Health Total: 

134 

Nature Total: 

92 

Magazine Title 

# Collected 

Magazine Title 

# Collected 

Human Rights 

42 

Astronomy 

10 

Newsweek 

50 

Nature 

50 

The Atlantic 

25 

Popular Science 

29 

The Progressive 

22 

Science 

119 

Time 

50 

Science and Children 

5 



Science Illustrated 

30 



Smithsonian 

38 





Politics Total: 

189 

Science Total: 

281 

Magazine Title 

# Collected 

Magazine Title 

# Collected 

Bicycling 

22 

Aviation Week 

45 

Car and Driver 

29 

PC World 

50 

Golf Digest 

26 

Popular Mechanics 

34 

Golf World 

24 

Technology Review 

17 

Runner’s World 

22 

Wired 

45 

Sporting News 

44 



Sports Illustrated 

50 







Sports Total: 

217 

Technology Total: 

191 
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(d) (e) (f) 



(g) (h) (i) 



G) (k) (1) 

Fig. 13: The 12 color-word topics produced by the model are visualized in this figure. The top panel 
shows the color histogram, the second and third color panels show the top two color palettes we 
extracted from this histogram. The word topics are visualized in the bottom panel as word clouds, 
with the size of a word being proportional to its weight. 
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Fig. 14: Topics vs. titles. The proportion of each of the 12 color-word topics, ki to ki 2 (see Fig. 5) 
for each magazine title including all the issues in the dataset from year 2000 to 2013 is illustrated. 
Note that the colors here are just legends for the purpose of visualization, and are not related to the 
color-word topics. 


ACM Transactions on Computer-Human Interaction, Vol. V, No. N, Article A, Publication date: January YYYY. 






















































































































Colors —Messengers of Concepts: Visual Design Mining for Learning Color Semantics 


A:27 


mr 


81 0.11 0.: 

62 0.22 0.1 


(a) 


2.21 0.12 1.68 0.30 0.32 0.48 0.28 0.34 0.23 0.77 0.50 0.19 

0.90 0.62 

0.72 0.95 


69 0.48 0. 


«iii w2]i 


w7|| w8ir 


w<i wiiin; 


2.30 0.31 1.52 0.41 1.04 0.57 0.70 0.33 0.17 0.96 0.38 0.55 


29 0.95 0. 


(b) 



(c) 


(d) 


linr 



w2|| 





1 

an 




w9|| wfll Willi wH 










ar 





1 

ar 


w 



Vk 

m 

cl 

1.98 

0 

69 1 

29 

0 

50 

0.30 

0 

35 0 

24 

0.43 

0 

48 0 

97 

0 

34 

0.17 


2.44 

0 

48 1 

43 

0 

38 

1 

10 0 

26 

0 

50 

0.57 

0 

18 

0 

90 

0 

30 

0 

48 

c2 

0.65 

0 

30 0 

76 

0 

47 

1.27 

0 

20 0 

42 

1.77 

1 

73 0 

84 

0 

58 

1.74 

C2| 

0.00 

0 

81 0 

91 1 

03 

0 

92 0 

50 

0 

34 

1 

21 1 

89 

0 

46 

0 

67 1 

34 

c3 

0.12 

1 

17 1 

46 

0 

79 

0.97 

0 

74 1 

19 

0.74 

0 

37 0 

94 

0 

86 

0.52 

C3j 

0.25 

0 

92 1 

42 

0 

60 

1 

33 0 

85 

1 

75 

0 

16 

0 

42 

0 

87 1 

65 

0 

29 

c4 

0.00 

1 

69 1 

16 

1 

13 

0.95 

1 

08 1 

64 

0.08 

0 

43 1 

04 1 

38 

0.22 

c4 

0.49 

1 

47 0 

73 

1 

26 

0 

75 0 

95 

1 

60 

0 

18 

0 

32 

1 

10 

0 

61 

0 

83 

c5 

0.47 

0 

84 1 

54 

1 

14 

1.42 

1 

14 1 

45 

0.36 

0 

23 1 

24 

1 

22 

0.74 

csi 

0.41 

1 

15 1 

69 

0 

72 

1 

28 1 

42 

0 

44 

0 

57 

0 

44 

0 

72 

2 

13 

0 

76 

c6 

0.81 

1 

15 0 

93 

0 

70 

0.65 

1 

75 0 

92 

0.28 

0 

12 0 

64 

0 

74 

0.74 

C61 

0.24 

1 

16 1 

20 

0 

29 

0 

77 1 

93 

1 

53 

0 

21 

0 

00 

0 

57 

1 

35 

0 

29 

c7| 

0.15 

1 

27 1 

05 

1 

11 

1.20 

0 

79 1 

82 

0.43 

0 

40 0 

35 

1 

32 

0.53 


0.15 

1 

18 0 

44 1 

53 

0 

80 1 

66 

1 

52 

0 

19 

0 

95 

0 

75 

1 

18 

0 

64 


0.09 

0 

27 0 

89 

0 

43 

1.53 

0 

71 0 

39 

1.93 

1 

04 1 

03 

0 

52 

1.27 

c8^ 

1.03 

0 

90 1 

42 

0 

83 1 

44 1 

38 

0 

42 

0 

69 

0 

00 1 

37 

0 

75 

0 

85 

c9 

0.56 

0 

00 0 

20 

0 

20 

1.42 

0 

10 0 

20 

1.87 

1 

95 0 

49 

0 

47 

1.23 

csi 

0.00 

0 

15 0 

39 

0 

39 

1 

30 0 

14 

0 

57 

1 

92 

2 

34 

0 

48 

0 

16 

1 

08 

cia 

0.60 

0 

42 0 

81 

0 

66 

1.71 

0 

49 0 

00 

1.42 

0 

97 1 

40 

0 

53 

0.90 

m 

0.90 

0 

52 0 

51 

0 

20 

2 

39 0 

15 

0 

34 

1 

24 

1 

53 

1 

43 

0 

26 

0 

41 

cll 

0.45 

1 

70 0 

64 

1 

31 

0.52 

1 

40 1 

02 

0.38 

0 

43 0 

65 

1 

24 

0.33 

clll 

0.00 

1 

27 1 

13 

1 

02 

0 

41 1 

14 

1 

37 

0 

00 

0 

00 

0 

86 

1 

31 

0 

00 

cl2 

0.89 

0 

51 1 

26 

1 

02 

0.55 

1 

49 0 

71 

0.66 

0 

39 1 

20 

0 

93 

0.61 

UiJ 

0.44 

0 

40 1 

28 1 

21 

0 

32 0 

45 

1 

14 

0 

43 1 

36 

0 

46 

0 

78 

1 

00 


(e) 


(f) 



■ iHl wi ll Wil l wal l W4|| ws ll we ll wtI I wal l wel l w^ | wii ll m 

Cl 2.10 0.40 1.40 0.31 0.53 0.33 0.29 0.40 0.30 0.84 0.45 0.30 

c2 0.61 0.30 0.49 0.71 1.21 0.34 0.38 1.75 1.97 0.76 0.62 1.41 

c3 0.15 1.27 1.51 0.82 0.99 0.97 1.05 0.53 0.31 0.58 1.04 0.33 

C4 0.04 1.67 1.12 1.25 0.66 1.20 1.73 0.12 0.28 0.63 1.51 0.39 

c5 0.42 1.29 1.50 1.11 1.45 1.10 1.67 0.42 0.38 0.93 1.05 0.52 

c6 0.74 1.02 1.04 0.66 0.56 1.88 0.94 0.23 0.12 0.59 1.11 0.42 

c7 0.13 1.36 1.25 1.03 1.04 0.97 1.66 0.30 0.44 0.44 0.98 0.41 

C8 0.48 0.29 0.55 0.47 1.35 0.42 0.34 1.80 1.41 1.02 0.46 1.00 

c9 0.47 0.04 0.13 0.28 1.29 0.08 0.12 2.00 2.04 0.44 0.41 1.24 

clO 0.99 0.28 0.69 0.52 1.80 0.35 0.22 1.51 1.07 1.42 0.59 1.14 

cll 0.27 1.66 0.99 1.33 0.39 1.18 1.58 0.26 0.27 0.69 1.40 0.42 

cl2 0.98 0.54 1.51 0.98 0.85 1.61 0.51 0.61 0.65 1.09 1.07 0.58 



(i) 


G) 



Fig. 15: Relevance matrices and for the first and second set of questions, respectively, com¬ 
puted for: (a) and (b) females, (c) and (d) males, (e) and (f) Non-US participants, (g) and (h) US 
participants, (i) and (j) Non-designers, and (k) and (1) designers. 
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Table II: Handcrafted Stop Word List. Note that these words are inspected and excluded by manually 
visiting the first 30 words in the word topics inferred by the model. 


day 

inside 

meaning 

reveals 

things 

winning 

minute 

dos 

issue 

meet 

romney 

tips 

frances 

colors 

double 

Jennifer 

minutes 

ryan 

today 

small 

grows 

easy 

joe 

month 

share 

top 

japan 

green 

essence 

Johns 

nation 

shows 

trick 

china 

simple 

exclusive 

Julie 

needed 

special 

ultimate 

stuff 

autumn 

eye 

kate 

obama 

Steve 

undos 

pages 

design 

faces 

klein 

ons 

stop 

ups 

mitt 


falling 

losing 

picks 

stories 

wanted 

week 


free 

lost 

preview 

summer 

ways 
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good 

makeover 

rated 

takes 

white 

super 


great 

making 

real 

talking 

work 

perfect 


guide 

matter 

red 

tells 

year 

spring 
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